{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a98030af-fcd1-4d63-a36e-38ba053498fa",
   "metadata": {},
   "source": [
    "# A full business solution\n",
    "\n",
    "## Now we will take our project from Day 1 to the next level\n",
    "\n",
    "### BUSINESS CHALLENGE:\n",
    "\n",
    "Create a product that builds a Brochure for a company to be used for prospective clients, investors and potential recruits.\n",
    "\n",
    "We will be provided a company name and their primary website.\n",
    "\n",
    "See the end of this notebook for examples of real-world business applications.\n",
    "\n",
    "And remember: I'm always available if you have problems or ideas! Please do reach out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d5b08506-dc8b-4443-9201-5f1848161363",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "# If these fail, please check you're running from an 'activated' environment with (llms) in the command prompt\n",
    "\n",
    "import os\n",
    "import requests\n",
    "import json\n",
    "from typing import List\n",
    "from dotenv import load_dotenv\n",
    "from bs4 import BeautifulSoup\n",
    "from IPython.display import Markdown, display, update_display\n",
    "from openai import OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc5d8880-f2ee-4c06-af16-ecbc0262af61",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize and constants\n",
    "\n",
    "load_dotenv()\n",
    "api_key = os.getenv('OPENAI_API_KEY')\n",
    "\n",
    "if api_key and api_key.startswith('sk-proj-') and len(api_key)>10:\n",
    "    print(\"API key looks good so far\")\n",
    "else:\n",
    "    print(\"There might be a problem with your API key? Please visit the troubleshooting notebook!\")\n",
    "    \n",
    "MODEL = 'gpt-4o-mini'\n",
    "openai = OpenAI()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "106dd65e-90af-4ca8-86b6-23a41840645b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# A class to represent a Webpage\n",
    "\n",
    "# Some websites need you to use proper headers when fetching them:\n",
    "headers = {\n",
    " \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36\"\n",
    "}\n",
    "\n",
    "class Website:\n",
    "    \"\"\"\n",
    "    A utility class to represent a Website that we have scraped, now with links\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, url):\n",
    "        self.url = url\n",
    "        response = requests.get(url, headers=headers)\n",
    "        self.body = response.content\n",
    "        soup = BeautifulSoup(self.body, 'html.parser')\n",
    "        self.title = soup.title.string if soup.title else \"No title found\"\n",
    "        if soup.body:\n",
    "            for irrelevant in soup.body([\"script\", \"style\", \"img\", \"input\"]):\n",
    "                irrelevant.decompose()\n",
    "            self.text = soup.body.get_text(separator=\"\\n\", strip=True)\n",
    "        else:\n",
    "            self.text = \"\"\n",
    "        links = [link.get('href') for link in soup.find_all('a')]\n",
    "        self.links = [link for link in links if link]\n",
    "\n",
    "    def get_contents(self):\n",
    "        return f\"Webpage Title:\\n{self.title}\\nWebpage Contents:\\n{self.text}\\n\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e30d8128-933b-44cc-81c8-ab4c9d86589a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ed = Website(\"https://edwarddonner.com\")\n",
    "ed.links"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1771af9c-717a-4fca-bbbe-8a95893312c3",
   "metadata": {},
   "source": [
    "## First step: Have GPT-4o-mini figure out which links are relevant\n",
    "\n",
    "### Use a call to gpt-4o-mini to read the links on a webpage, and respond in structured JSON.  \n",
    "It should decide which links are relevant, and replace relative links such as \"/about\" with \"https://company.com/about\".  \n",
    "We will use \"one shot prompting\" in which we provide an example of how it should respond in the prompt.\n",
    "\n",
    "This is an excellent use case for an LLM, because it requires nuanced understanding. Imagine trying to code this without LLMs by parsing and analyzing the webpage - it would be very hard!\n",
    "\n",
    "Sidenote: there is a more advanced technique called \"Structured Outputs\" in which we require the model to respond according to a spec. We cover this technique in Week 8 during our autonomous Agentic AI project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6957b079-0d96-45f7-a26a-3487510e9b35",
   "metadata": {},
   "outputs": [],
   "source": [
    "link_system_prompt = \"You are provided with a list of links found on a webpage. \\\n",
    "You are able to decide which of the links would be most relevant to include in a brochure about the company, \\\n",
    "such as links to an About page, or a Company page, or Careers/Jobs pages.\\n\"\n",
    "link_system_prompt += \"You should respond in JSON as in this example:\"\n",
    "link_system_prompt += \"\"\"\n",
    "{\n",
    "    \"links\": [\n",
    "        {\"type\": \"about page\", \"url\": \"https://full.url/goes/here/about\"},\n",
    "        {\"type\": \"careers page\": \"url\": \"https://another.full.url/careers\"}\n",
    "    ]\n",
    "}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b97e4068-97ed-4120-beae-c42105e4d59a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(link_system_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8e1f601b-2eaf-499d-b6b8-c99050c9d6b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_links_user_prompt(website):\n",
    "    user_prompt = f\"Here is the list of links on the website of {website.url} - \"\n",
    "    user_prompt += \"please decide which of these are relevant web links for a brochure about the company, respond with the full https URL in JSON format. \\\n",
    "Do not include Terms of Service, Privacy, email links.\\n\"\n",
    "    user_prompt += \"Links (some might be relative links):\\n\"\n",
    "    user_prompt += \"\\n\".join(website.links)\n",
    "    return user_prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bcbfa78-6395-4685-b92c-22d592050fd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(get_links_user_prompt(ed))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a29aca19-ca13-471c-a4b4-5abbfa813f69",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_links(url):\n",
    "    website = Website(url)\n",
    "    response = openai.chat.completions.create(\n",
    "        model=MODEL,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": link_system_prompt},\n",
    "            {\"role\": \"user\", \"content\": get_links_user_prompt(website)}\n",
    "      ],\n",
    "        response_format={\"type\": \"json_object\"}\n",
    "    )\n",
    "    result = response.choices[0].message.content\n",
    "    return json.loads(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74a827a0-2782-4ae5-b210-4a242a8b4cc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Anthropic has made their site harder to scrape, so I'm using HuggingFace..\n",
    "\n",
    "huggingface = Website(\"https://huggingface.co\")\n",
    "huggingface.links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3d583e2-dcc4-40cc-9b28-1e8dbf402924",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_links(\"https://huggingface.co\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d74128e-dfb6-47ec-9549-288b621c838c",
   "metadata": {},
   "source": [
    "## Second step: make the brochure!\n",
    "\n",
    "Assemble all the details into another prompt to GPT4-o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "85a5b6e2-e7ef-44a9-bc7f-59ede71037b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_all_details(url):\n",
    "    result = \"Landing page:\\n\"\n",
    "    result += Website(url).get_contents()\n",
    "    links = get_links(url)\n",
    "    print(\"Found links:\", links)\n",
    "    for link in links[\"links\"]:\n",
    "        result += f\"\\n\\n{link['type']}\\n\"\n",
    "        result += Website(link[\"url\"]).get_contents()\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5099bd14-076d-4745-baf3-dac08d8e5ab2",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(get_all_details(\"https://huggingface.co\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9b863a55-f86c-4e3f-8a79-94e24c1a8cf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"You are an assistant that analyzes the contents of several relevant pages from a company website \\\n",
    "and creates a short brochure about the company for prospective customers, investors and recruits. Respond in markdown.\\\n",
    "Include details of company culture, customers and careers/jobs if you have the information.\"\n",
    "\n",
    "# Or uncomment the lines below for a more humorous brochure - this demonstrates how easy it is to incorporate 'tone':\n",
    "\n",
    "# system_prompt = \"You are an assistant that analyzes the contents of several relevant pages from a company website \\\n",
    "# and creates a short humorous, entertaining, jokey brochure about the company for prospective customers, investors and recruits. Respond in markdown.\\\n",
    "# Include details of company culture, customers and careers/jobs if you have the information.\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6ab83d92-d36b-4ce0-8bcc-5bb4c2f8ff23",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_brochure_user_prompt(company_name, url):\n",
    "    user_prompt = f\"You are looking at a company called: {company_name}\\n\"\n",
    "    user_prompt += f\"Here are the contents of its landing page and other relevant pages; use this information to build a short brochure of the company in markdown.\\n\"\n",
    "    user_prompt += get_all_details(url)\n",
    "    user_prompt = user_prompt[:5_000] # Truncate if more than 5,000 characters\n",
    "    return user_prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd909e0b-1312-4ce2-a553-821e795d7572",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(get_brochure_user_prompt(\"HuggingFace\", \"https://huggingface.co\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e44de579-4a1a-4e6a-a510-20ea3e4b8d46",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_brochure(company_name, url):\n",
    "    response = openai.chat.completions.create(\n",
    "        model=MODEL,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": get_brochure_user_prompt(company_name, url)}\n",
    "          ],\n",
    "    )\n",
    "    result = response.choices[0].message.content\n",
    "    display(Markdown(result))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e093444a-9407-42ae-924a-145730591a39",
   "metadata": {},
   "outputs": [],
   "source": [
    "create_brochure(\"HuggingFace\", \"https://huggingface.com\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61eaaab7-0b47-4b29-82d4-75d474ad8d18",
   "metadata": {},
   "source": [
    "## Finally - a minor improvement\n",
    "\n",
    "With a small adjustment, we can change this so that the results stream back from OpenAI,\n",
    "with the familiar typewriter animation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "51db0e49-f261-4137-aabe-92dd601f7725",
   "metadata": {},
   "outputs": [],
   "source": [
    "def stream_brochure(company_name, url):\n",
    "    stream = openai.chat.completions.create(\n",
    "        model=MODEL,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": get_brochure_user_prompt(company_name, url)}\n",
    "          ],\n",
    "        stream=True\n",
    "    )\n",
    "    \n",
    "    response = \"\"\n",
    "    display_handle = display(Markdown(\"\"), display_id=True)\n",
    "    for chunk in stream:\n",
    "        response += chunk.choices[0].delta.content or ''\n",
    "        response = response.replace(\"```\",\"\").replace(\"markdown\", \"\")\n",
    "        update_display(Markdown(response), display_id=display_handle.display_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56bf0ae3-ee9d-4a72-9cd6-edcac67ceb6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "stream_brochure(\"HuggingFace\", \"https://huggingface.co\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87bd1188",
   "metadata": {},
   "outputs": [],
   "source": [
    "stream_brochure(\"HuggingFace\", \"https://huggingface.co\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9e7375d",
   "metadata": {},
   "source": [
    "## **Multi-lingual with Multi-Tone in Desire Format**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "af5c959f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def multi_lingual_stream_brochure(company_name, url, language, tone):\n",
    "\n",
    "    system_prompt = f\"\"\"\n",
    "You are an assistant that analyzes the contents of several relevant pages from a company website and creates a visually appealing and professional short brochure for prospective customers, investors, and recruits. \n",
    "The brochure should be written in {language} and use a {tone.lower()} tone throughout.\n",
    "\n",
    "The brochure should follow this structure (in {language}):\n",
    "\n",
    "1. **Front Cover**:\n",
    "   - Prominently display the company name as Title.\n",
    "   - Include a compelling headline or tagline.\n",
    "   - Add something engaging relevant to the company’s mission.\n",
    "\n",
    "2. **About Us**:\n",
    "   - Provide a brief introduction to the company.\n",
    "   - State the company’s core mission and vision.\n",
    "   - Mention the founding story or key milestones.\n",
    "\n",
    "3. **What We Offer**:\n",
    "   - Summarize the company's products, services, or solutions.\n",
    "   - Highlight benefits or unique selling points.\n",
    "   - Include testimonials or case studies if available.\n",
    "\n",
    "4. **Our Culture**:\n",
    "   - Outline the company’s key values or guiding principles.\n",
    "   - Describe the workplace environment (e.g., innovation-driven, inclusive, collaborative).\n",
    "   - Highlight community engagement or CSR initiatives.\n",
    "\n",
    "5. **Who We Serve**:\n",
    "   - Describe the target customers or industries served.\n",
    "   - Mention notable clients or partners.\n",
    "   - Include testimonials or endorsements from customers.\n",
    "\n",
    "6. **Join Us**:\n",
    "   - Detail career or internship opportunities.\n",
    "   - Highlight benefits, career growth, or training opportunities.\n",
    "   - Provide direct links or steps to apply.\n",
    "\n",
    "7. **Contact Us**:\n",
    "   - Provide the company’s address, phone number, and email.\n",
    "   - Include links to social media platforms.\n",
    "   - Add a link to the company’s website.\n",
    "\n",
    "8. **Closing Note**:\n",
    "   - End with a thank-you message or an inspirational note for the reader.\n",
    "   - Add a call-to-action (e.g., “Get in touch today!” or “Explore more on our website”).\n",
    "\n",
    "Ensure the content is concise, engaging, visually clear, and tailored to the target audience. Use headings and subheadings to make the brochure easy to navigate. Include links and contact information wherever applicable.\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "    \n",
    "    stream = openai.chat.completions.create(\n",
    "        model=MODEL,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": get_brochure_user_prompt(company_name, url)}\n",
    "          ],\n",
    "        stream=True\n",
    "    )\n",
    "    \n",
    "    response = \"\"\n",
    "    display_handle = display(Markdown(\"\"), display_id=True)\n",
    "    for chunk in stream:\n",
    "        response += chunk.choices[0].delta.content or ''\n",
    "        response = response.replace(\"```\",\"\").replace(\"markdown\", \"\")\n",
    "        update_display(Markdown(response), display_id=display_handle.display_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "744bfc05",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "multi_lingual_stream_brochure(\"OpenAI\", \"https://openai.com/\", \"Urdu\", \"humorous, entertaining, jokey\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
